chi-bench: U2Med · U2MedFellow · U2Med - #22

Hi @ZhongZhi-Qiang, @gechengze — thank you for the submission and for your patience while we reviewed it.

We found several things we cannot reconcile with the packet contract or the leaderboard's pass@1 protocol. We would like to give you the chance to clarify before we make a decision. Would you answer the following questions?

1. Model identity

submission.json lists the model as U2Med, but every result.json records config.agent.model_name as openai/qwen3.5_27b_sft_0609 (cm and part of pa_um, run 2026-07-19) or openai/qwen3.5_27b_sft_0620 (pa_provider and the rest of pa_um, run 2026-08-06 and 2026-08-21).

• Could you describe the base model and the fine-tuning that produced these two checkpoints?
• Was any χ-Bench task content (instructions, fixtures, expectations, solutions, or trajectories from the public dataset) used in the SFT data?
• Which single checkpoint do you consider to be "the" submitted system?

2. How the pa_um trials were assembled

The 25 pa_um trials come from four separate Harbor jobs:

Within a Harbor job each trial start normally coincides with another trial's finish. In job 07050be1 four included trials (t014, t024, t026, t035) start at times that match no included finish, and the same is true for t019 in job babfe0f6. That pattern indicates trials that ran in those jobs but are not in the packet.

• Could you explain how the 25 trials were selected, and share the complete trial lists (with rewards) for all four jobs?
• The leaderboard scores a single run with n_attempts: 1, as your sub.yaml states. Combining trials from several runs and checkpoints is not compatible with that, so we would need a fresh single run in any case (see "What we would need" below).

3. Prompt content beyond instruction.md

The first user message in each trajectory contains the task's instruction.md followed by additional text:

• In all 25 pa_provider trials, an identical block of about 15,500 characters.
• In each of the 12 pa_um trials from the 0620 runs, a different block of 11,000 to 15,000 characters. Section titles include, for example, PA-UM PET MPI P2P UPHOLD ROUTE (t016), PA-UM HYPOGLOSSAL/DISE P2P RESOLUTION with "Use overturn_to_approval" (t019), PA-UM HOPD SITE-OF-SERVICE DECISION with "Submit an MD deny" (t021), and PA-UM VIRTUAL UPPER-GI P2P UPHOLD ROUTE (t036). The t013 block quotes the IQ-OR-001 to IQ-OR-005 criterion texts verbatim and the specific patient's PT evaluation values (ODI 48%, NPRS 7/10 and 9/10, lumbar flexion 35°, extension 10°).
• The pa_provider block contains procedure-specific rules such as "CPT 74178 … choose do_not_submit", "hybrid cervical surgery … choose do_not_submit", and "A 24/48-hour Holter is not a default match … record not_indicated and choose do_not_submit". Those match the expected outcomes of t036, t032, and t015 respectively.
• The 0609 trials (2026-07-19) use the unmodified instruction.md.

Could you explain how these prompt blocks were produced, whether they were written per task, and whether fixtures/expectations.json, solution/, or tests/ from the public dataset were consulted when writing them? Generic workflow guidance is fine under the rules; task-specific expected outcomes are not, so we need to understand which of the two this is.

4. Care-management trials ran without a working Anthropic key

In every one of the 25 cm trials, each cm_outreach_send_message call returned:

Patient simulator failed: ANTHROPIC_API_KEY is not set or invalid.

No patient turn ever occurred. In 22 trials the agent then called end_outreach_call with a summary recording, for example, consent_obtained: true and a 15 to 25 minute call duration. The verifier phase lasted about 3 seconds per trial, and all judge.cm.* checks are not_applicable, which means the claude-opus-4-7 judge named in provenance.json did not run for this domain. On the main branch, cm verification takes roughly 90 to 320 seconds and produces True/False verdicts.

• Was ANTHROPIC_API_KEY available inside the container during the 2026-07-19 cm run?
• As recorded, the cm rewards were computed from the deterministic checks only. We cannot accept those as judged scores.

5. Peer-to-peer turns

In the four p2p_payer trials, every p2p_session.send_turn call returned HTTP 403 Request not allowed, so no counterpart turn took place. The trial config includes CHI_BENCH_PATIENT_SIM_BASE_URL=https://api.gpugeek.com. Could you describe how Anthropic traffic was routed for these runs?

We will hold this PR at needs-review until we hear back. If any of the above turns out to be a harness or documentation problem on our side, we would genuinely like to know so we can fix it, and we are happy to help you get a clean run through. We can look at the logo request once the submission itself is settled.

Thanks again for engaging with the benchmark.